MASSACHUSETTS INSTITUTE OF TECHNOLOGY 
PROJECT MAC 


Artificial Intelligence 
Memo No * 171 


DECOMPOSITION OF A VISUAL SCENE 
INTO THREE-DIMENSIONAL BODIES 


MAC-M-391 
January 1969 


Adolfo Guzman 


ABSTRACT 


The program described here takes as its input a collection 
of lines, vertices and surfaces describing a scene, and 
analyses the scene into a composition of three-dimensional 
objects. 

The program does not need to know the form {model, or 
pattern) of the objects which are likely to appear* the scene 
is not searched for cubes, wedges or houses, with an a-priori 
knowledge of the form of these objects? rather, the program 
pays attention to configurations o£ surfaces and lines which 
would make plausible thr ee-dimensional sal ids, and in this way 
'bodies' are identified. 

Partially occluded bodies are handled correctly. The pro¬ 
gram is restricted to scenes formed by straight lines, where no 
shadows or noise are present, it has been tested in rather 
complicated scenes, composed by rather simple objects. Examples 


are given 
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INTRODUCTION 

We eo-nflUIer visual scenes composed by the optical 
image of a group o i bodies. When Bitch a scene is “seen” 
by a ewnptlter through a film spot scanner, image dia- 
sectiir, Of similar device, it can be treated aa a two-di- 
ri:eri*ii.icuk! umi.v of numbers, or as a function of two 
v*ri*bles. 

At a higher level, u> accnc could be meaningfully de¬ 
scribed iis ft eonglomcrate of points, Lines and surfaces, 
-with properties foCCrdinstcs, slopes, ,.) attar-bed to 
them. 

StiEl a more Sophisticated description could use terms 
Opoeemmg thfc tithes Of objects whith compose such a 
*oene 3 indies, ting their portions, inter-rala-tLons, etc. 

This pfcpw- describes * program which finds bodies in 
* scene, presumably formed by three-dimensional ob¬ 
ject*. StKDOor them may not be completely visible. The 
picture it presented as a line drawing. 

When SEE—the pretentioua name of the program-- 
ttmlytes the scene TRIAL {see .Figure 1 'TRIAL 1 ), the 
results aw; 

(BODY 1.13 A H rl> 

(BODY 2- IS :11 ;I3 flO) 

(BODYa.ie A rS J =9:13) 

SEE looks for threc^-dimcnsional objects in a two- 
dimensional scene. Tht'sCcaC itself is not obtained from 
a visual input device, or from an array of intensities or 
brightii*ss. Rather, it is assumed that a preprocessing 
of soma sort has taken, place, and the scene to be ana¬ 
lysed ia available in a symbolic format- (to be described 
in a later Section), in terms of pnuata (vertices}, fine* 
(edges), and surfaco (regions). 

SEE docs not have a pronconcieved idea of the form nr 
model of the objects which could appear in a given 


Beene. The only supposition is timt the bodies are solid 
objects formed by plane surfaces; in this way, it can 
not find "cubes" or "houses” in * scene, since it does 
not know what a "house" ia. OnoeSEE baS partitioned 
a scene into bodies, some other program will woik on 
thorn and decide which of those bodies are ‘'houses." 

Thus, &EE ia intended to serve h.m a link between a 
pre-processor which transforms intensity pictures 
into point or fine pictures, * and a rocogftiJter (such as 
TD 1 or DT 1 }, which handles this line picture and 
finds bodies, objects or zones matching with certain 
patterns or models. Instead of searching through the 
whole acme looking fee parts to match its models, the 
work of the rccognuer becomes simpler after SEE has 
partitioned the acene into bodies, because the data to be 
searched (matched) are smaller and better organized. 

The analysis which SEE makes of the different, scenes 
generally agrees with human ■pinion, although in some 
ambiguous cases it behaves rather conservatively. Dis¬ 
tributed over these pages, the reader will find examples 
of scenes analysed by SEE, and the peculiarities and 
bch&vlor of the program will become clear. 

The program SEE, written in LISP, has been tested 
in the PDP-d machine of the Artificial Infdligcncs 
Group, Project MAC, at Massachusetts Institute of 
Technology, A preliminary version, written in CON¬ 
VERT, * #'M used eatensivdy for * quick test of ideas 
which shaped the program, to its Mtued form. The analy¬ 
sis of a scene lakes from 3Qlo 90 seconds, with the pro¬ 
gram running interpreted under the interpreter of the 
LISP programming system. 

A more technical description of SEE can be found in 
an unpublished memorandum. T 

Refuted work 

Rudd H. Canaday 1 in ltHj3 nosdyisd seenes Mm- 
posed of two-dimsiuiiojuil overiufipfog objects* “straight- 
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S—TRIAL 

Th» [iTAgrtrii Hiafjiw thin pud( flu* U bwiti: 

(BODY I IS 6 (i :l) 

{BODY 2 It! ill :12:10> 

1 (body marts a :r a.:» nil 

aided piece* of oardbOMd ■" Hii program breaksthe image 
into LLfl component pout* (tht pieces of cardboard), do- 
■aibea each give* the dOptb of each part in the 
image (or #eeue), Mid state* which parts cover which. 

Roberta ["] in ifieS described pno^fanis that {1) con¬ 
vert a picture (a scene) into aline drawing and (S) pro¬ 
duce a ittre*^imei3flk)iia] description of the objects 
shown in the drawing in terms of models and their 
tranaforruatkais. The main restriction Oft the line* is 
that they should he # perspective projection of the sur¬ 
face boundaries- ml a set of three-dinwiwonal object* 
with planar enu-Jaces. He retie* -00 perspective and 
numerical computations, while SEE uaea a heuristic and 
symbolic (i.B., non-numerical ) approach. Also, SEE 
does not need models to Isolate bodLea. Roberta' work I s ? 
probably the moat important and closest to ours. 

Actually, several research groups (at MaasachusettB 
Institute of Technology^ l> at- Stanford L'niveradty,, " 
at Sttofori Research Institute u ) work actively to¬ 


wards the resbratron o-f a mechanical manipulate, t.c, r 
an intelligent automata who could visually perceive and 
successfully Interact with its enviornment, under the 
control of a computer.. Naturally, the mechanisation of 
visual perception forma part of their research, and im¬ 
portant work begins to emerge from them in ttda area. 

Organization of (he paper 

it-^formed by the follow inj[ beading: 

* Introduction and related previous work. 

* Input Format, The representation of the scene an 
it is entered to the computer. 

- Format of a Scene. The representation of the scene 
as SEE expects. 

■ Type (rf Vertices, daseiheation of vertices ac¬ 
cording to their topology. 

* The program. Analysis of the algonfhm, descrip¬ 
tion of heuristics. 

* Interesting; examples. Discuwdon- Future work- 
Jjipiil/ormoi 

For testing purposes,. the scenes are entered by bind 
in a simplified format (called input- format), and then 
some routines convert this to the form required by SEE. 
Eventually , the data will come from a visual input de¬ 
vice, through a preprocessor , 1, * 

EiampleKiflfaacene 

Suppose we want to describe the scene 'CUBE. 3 We 
begin by giving (inLISP}aValueto f CXIRE-' (See Figure 
2 'Cube') 

(SETQCUBE (QUOTE (A 1.01,0(:1B AO) 

B Ij0 £j 0 (:1 E ;S C ;4 A} 
C J.07.0 (iD ;4 U) 

D 0.0 7-0 (S E F :4 C) 
E 6-0 5 0 (:2 B .1 Q 3 D) 
F 8.030(30 AH) 

G 6-0 1.0 (11 A :4 F :3 E> ? ' 

))) 

Thus wc asEocdate with each vertex its coordinate* 
and a list, m counterclockwise order, of region* and ver- 
ticra radiating from that vertex. 

Tl^i conversion of the arena, aa just given, into the 
form which 5EE ejects, is marie by the function 
LLENA; thus, (LLENA CUBE) wiB put in the prop- 
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PIGUflE 3—'OWE. 1 A s-MOft. Verticw aTtd uurJacW trsdismi!' 
are die rrktia raimpnrumta, of :* w*ne. 


irty Alt of CUBE the properties REGIONS end 
VERTICES; in the property list of each vertex, t-ho 
propcriiwXCOR, YCQR, NREQION0, N VERTICES 
end KIND ere placed; and in the property list of c&ch 
region, it places the properties NEIGHBORS and 
KYERTICE8. It also marks region :4 aa pan of the 
background. 

In other words, LLENA converts a scene from the 
'Input Format" to the 'Format of a Scene' desprihed in 
tbr opst section ■ 

Fermat of a tM) 

A scene is presented to onr program as a scene in a 
special symbolic format-, which, we now describe, 
Essentially, it is an a'rangemerit of relations between 
vertices and regions. 

A scene has a name which identifies R- tills ii&me is an 
atom whose property list contaioa th* prtperUee 
"REGIONS," 'VERTICES" and 'BACKGROUND.' 
For example, the scene ONE {see Figure 3 "ONE') has 
the name "ONE." In the pnpert-y list of "ONE" we find 

REGIONS —(:I 3 3 j# =5 ;6) 

Unordered list of regions composing 
soenaONR. 

VERTICES —CAHCDEFGHIJK> 

Unordered list of vertices com¬ 
posing the scene ONE. 

BACKGROUND— <jfl) 

UfLordcrcd list of region compos¬ 
ing t^C back*round of tbv scene 
ONE- 


Ergton 

A region corresponds to <t SUnfAcc limited by a simple 
connected ourvy. For instance, ID ONE, the surface de- 
limitod by the vortices A H C ia » region, called rl, but 
GIJRDHifitot- 

E*eh region, has w name an Atom which posscMWaddi - 
tkrnaj properties describing different attributes of the 
region it) question. The**; are 'NEIGHBORS, 1 "KVER* 
TICES," and "FOOPV For the teflon In 

scone ONE formed by the AE, ED, DK, KC, CA, 
bus '^2" at its name, In the property list of 3 we Eio4; 

NEIGHBORS — (3 :4 £ ;l :6) 

Counterclockwise ordered list of 
ati regions which *rt neighbors to 
-.2. For each region, this list ll 
unique.Up to cyclic permutation, 

K VERTICES — (DEACK) 

Counterclockwise ordered list of 
aEE vertices which belong to the 
region :2- This list- is unique up to 
cyclic permutation. 

FOOT — (;3 0-4 E * All Gift K) 

Counterclockwise ordered list- of 
site mating neighbors and kve> 
tices of :2- This [ist La unique up to 
'* tycllc permutation. 

The FOOP property of a region is formed by a man 
who walks on its boundary always having this region to 
his left, and takes note of the regions to bis right and of 
the vertices which he finds in hie way. 
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Vertex 

A vertex ia the point where two or more Lines of the 
aoene meet; far instance. A. G. and K are vertices of the 
awns ONE. 

Each vertex has ae name an stem which. poBscsc? siddi- 
tigiutl properties deacribiog different attributee of the 
vertex in question, These are "XCOfE,' ‘YCOR,' 
'N VERTICES,' 'NREGIONS/ and 'KIND/ For 
example, tk vertex H {see ansae ONE) has in its prop¬ 
erty list r 


XCOR 

YCOR 

^VERTICES 

NRXGIONB 

KIND 


— a,o 

x coordinate 

— 1J iJQ 

V ■coordinate' 

— (IGD) 

Counterclockwise ordered List of 
vertices to which H is connected, 
This lift is unique up to cyclic 
permutation. 

— {3 :S ^4> 

CountemLocItwise ordered list of 
regions to which II belongs. 
This list u unique up to cyclic 
permutation, 

— (£1:50-413) _ 

CountercicHlcwise ordered list of 
after eating nregioru and n vertices 
of H. This hat ie unique up tn eye- 
tic permutation. 


Tbe KIND property of « vertex i* farmed by a man 
who stands at the vertex and, white rotating counter- 
ofookwise, taken note of t-tte regtona and vertices which 
ho sees . 

NREOIONS and NVERTICES are then easily de¬ 
rived from KIND: take the odd positioned olcmcnte 
of KIND, and its even-positioned elements, ir-speo- 
lively. 


Type* e/ e-tfriice a 

Vertices- are classified according to th* slops, disposi¬ 
tion and number of Lines which form them- The func¬ 
tion (TYPEGENERATGR h\ wh«e L is a list of vw- 
liew, performs this clanifiCfttiOP, putting in the prop- 
arty fill of each one ui the elements of L, the type to 
which it belongs- 

The TYPE of a vertex ia always a list of two ele¬ 
ments; the first is the Ijfpe-itama; -one of ( W 'FORK/ 
‘ARROWS T/ K/ %' PEAK- 'MULTI 3 ; the second 
cJomcnc ia the sfaiirPt, which fvnonlley is a list, whose 


form varies with the type-now and contains informa¬ 
tion in a determined order about-the vertex in question. 
{See Table I 'VERTICES^ 


TAULL 1 — ,| VEli,TICEi! r CIStHifaaUcii of vsttiwo of a jcme. 
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Thc pfjjpmitJ 

The program named SEE ueospte. a acene expressed 
in the notation described above and produces outputs 
lists identifying and describing the bodies present in the 
Boene- 

In fhia section wc describe the program, and how it 
achieves its goats, by discuasing the procedures, 
heuri«tacH etc,, employed and the way they work. 
We begin, with several examples- 
Example 1. Scene f STACK‘ This scene Figured 
'STACK') ia analysed by SEE, with the following re¬ 
sult*: 


(LOCAL ASSUMES f:5> {-13 *4) SAME BODY) 
(BODY I. IStl £ 3 :1B :17) 

(BODY 2. IS l*:1«:I4) 

(BODY 3, L& £7 * ill d2) 

(BODY 4, IS fl :3 -Id) 

(BODY 5-IS l13 ;14 :fy 
(BODY 6. IS ;B0 ^1G} 

i'lesuite forsfiene STACK 


Example 2. Scene 'BRIDGE/ With this cxwupl®, 
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r 



FIGURE 4—"STACK, 1 TW# K*n" if fnalyaad by SEE, wilt 
multi iUifil*j in Efamjife t. All hodLH fra rorrwtlr faimd. 
Snmn nf the vartim* ippMr in the dranrirj with thrir un»; in 
04!# drawing wii will orrJt etc uuinna; ire am also nmiring the 
c rc.rdinnt* ajno. 


we will give details- of the prOBUm 3 ! operation. We 
atari by loading Into LISP tlw prt£T*™iA iar SEE and 
ita auxiliary functicma, 

Then, we evaluate; 

{UREAP BRIDGE BCEN3) ft} 

Ttua causes bccm BRIDGE (aw 
Figures 'BRIIXJE'),.to bo fft*d 
fin the fomal &a*eribed pre- 
viuualy) and traiafonnaed to the 
pm per form which SEE expects 
I'aLsc' described before). 


Finally,, we evaluate 

(SEE (QUOTE BRIDGE)) 

This calls SEE to work nn. 

BRIDGE. 

Results appear in Table I t, 'RESULTS FOR BRIDGE. 1 


(LOCAL ASSUMES (:1B) (:1G) SAME BODY j 
(LOCAL ASSUMES (38) {;»} SAME BODY) 

(LOCAL ASSUMES (:I0) (;ft ill ifi sfl SAME BODY) 
(LOCAL ASSUMES (i7) (A ill AAA :M) SAME BODY) 

(BINGLEBODY ASSUMES (:1# :1&J (:IA) SAME BODY) 

RISULT3- 

(BODY 1. ES £i 0 :i!l £7:l3de) 

(BODY?. 10 m 30 S3) 

(BODY a, 10 :17 3 30) 

(BODY A. IS :C ^ RmiJta frw rana 'BRIDGE- 1 

(BOD YAIB ;L4 ;»:»> 

(BODYi, 10 m :]#:!*) 

(BODY 7. IS 39 33) 

(BODY A IS A ill ifl * it M J) 

TABLE If—RESULTS FOIL BIUDGE 

SEE linrj night bodjaa in hmqb 'BRIDGE' (S» figure S 
TOID.W) UnuMMsai^ detail him bon rarmaYtd from f-h# 
dnwiogi; bba ™dcr cmat bar in uund lilt tin aihijiUto M4Pf 
r,-.Ti<j.iru ra-tit™ iVhc« Tiurn wi da not appear In tb# lira wing?) 
and their onoidinniw (wtinh ate also nob nh.oeTi.1, *e w*U ff 
aod mptfu, 



FIGURE 5—‘BRIDGE. 1 The Ik* body 35 Ut :27 :11 3 :t2 a 
oormcily idenLiSsd. (Boo Table III 
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SEE and ice pari a 

The operation of SEE- is quite straightforward; the 

program, ia not recursive and does not -do any tree 
search. Its main parte, ■which are executed one utter 
another, unless otherwise indicated, am 

FTLtSCENE- The properties SLOP and TYPE- 
are generated for each vertex, if they were- not already 
present. 

CLEAN. Did properties used internally by SEE are 
removed from vertices and regions. 

EVIDENCES. An analysis ia made of vertices, regions 
and associated information, in search of ckos that imlioate 
that two regions form part of the same body, If evidence 
exists that two legions in fact belong to the same body* 
they arc Linked or marked with a ’ f gcnsym," {both re¬ 
ceive the same new Label). There are two kinds of links,, 
called strong (global) or weak (Local). 

LGCALEVIDENCE, Some features o£ the scene will 
weakly suggest that a group of regions should he con¬ 
sidered together, as part of the same body. This part of 
the program is that which produces the loul' links or 
evidence. 

GLOBAL. The 'strong' Links gallwred mt far are ana¬ 
lysed; regions are grouped into "iLUolei" of bodies, 
which grow until some conditidus fftU to he satisfied (a 
detailed explanation follows latter), 

LOCAL. Weak evidence ia taken into account for de¬ 
ciding which of the unsatisfactory global links should be 
considered satisfactory, and the corresponding nuclei of 
hod its are then joined to form a single and bigger 
nucleus, AtoUmptioM rfUkLe by LOCAL are printed 
(nee output, of SEE), LOCAL may tall GLOBAL again, 
or go on. 

SINGLEBQDY, If a single region does not belong to 
a larger nucleus, hut ia linked by one strong evident to 
auothet region, it is incorporated into the nucleus of 
that other recon. This part of the program, may eaSS 
GLOBAL or LOCAL again, if necessary, or continue. 
SINGLE BODY also prints its assumptions 

RESULTS). The regions belonging to the hPCkground 
are fLcreeiLfH.1 opt, and the results are printed, 

OntenKion 

Here is eaplaioad in considerable detail each of the 
parts of BEE that have been sketched shove. This will 
help the reader understand the behavior oE the program, 
its strength, and deficiencies. 

Exsmpb. Scene TOWER, 1 First, we begin hy show¬ 
ing s typical analysis of SEE with a somewhat eompLL- 
»ted scene (tee Figure fi J TOWER*). Mont of the areues 
contain several "nasty" M-incidenccs; a vertex of an ob¬ 
ject lies precisely on the edge of another object; two 
nearly parallel lines are merged into a single one. etc. 


This; has been done on purpose, since a non-so- 
pbintieoted preprocessor wiLL tend to make this kind of 
error. 

The output ia given in Table III. 'RESULTS 
FOE TOWER. 1 



FIGURE I?—TOWER-' Neither LOCAL nor SIJ-TXF.BODY 
ire- □koswM , j‘ to iXifiwUly pa™ thii scene hue th* tmdici which 
fioi it. There *es many 'rlrno^' Luik* In tliii socjie and SEC 
rp*k#fl gtuwi ilw of Lhmii (S« Tahln illj 


(BODY 1.10 3:2:1) 

(BODY 5, IS * :IS :i) 

(.BODY 3. 10 at :ITJ 
(BODY 4. IB fl iT *) 

(BODY 6. IS :« :11 3} Itoacltu for TOWER 

(BODY ft IB :I3 :U :lS) 

(BODY 7. IB la 33) 

(BODY ft. IS :30 :3fl SI) 

TABLE HI—RjjflijJt*. far Figure S TOWER 1 


FTLI£CENE, GLEAN. These two parte of BEE arc 
sample; if necessary, FTLLSCENE calls SLQPGENER- 
ATGR and TYFEGBNERATOR) GLEAN removes 
same unwanted properties, 

EVIDENCE. Strong or global links arc placed by 
this part oF the program. Two functions arc uesd; 
EVERTTCES and TJOINT9- 
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BYERTlGEfl. Thia fiMictiom considers Bach vertex of 
tire scene under the following rule*; 

L. Vertiee* of the type 'L 1 are not coneidontd, 

FORK if time meet at a vertex of 

the FORK type (Table IV" (h)), *nd 
none of them in -tbs background., link* 
between them will be formed. For instance 
iilt Figure A 'TOWER/ we satabliah tbe 
ttata :I&— 2Q, ;19— ;2l, and :2Q— :2l, 
Nevertheless, gome of these Linka may not 
be produced: in Figure 7 "FORK/ the liak 
between :3 and :2 Lj urf produced because 
Q Is fc "paBBnig T**:; the lint: between ;J «tid 
$ if not generated b«iHlBC R is an L.' The 
link between -1 and :2 « ganffrated- 

Thka heuristic is powerful, and (see Figure 
5 'BRIDGE 3 ) tdlowe us r for instance, 
wbLIs working on vertex jb r to omit the 
Uttk between regions r5 and £ i® scene 
'BRIDGE. 1 Nevertheless, this $ame heuiv 
«sKe puts a link between regiona # acid & 
of the Ed,me flcane, Ae we shall see later, 
this situation ia not too bad. 

ARROW. Thin type of vertex (Table tV (e)) causes 
two of ita repooa (the 'left' and the 
"right* one) to be Joined \ in Table IV (e), 
W6 will put a link between •! and % which 
OOUnfcs so evidence that :1 and 2 belong to 
the same body. Nothing is said about 

For instance, this type of vertex joint? 1; 
and 3 in Figure 5 'BRIDGE.* 

Xr Two cases are distinguished; 

(a) ThaX is formed, by tbe intersection of 
two line&. No evidence is formed. 

(b) Otherwise (Table IV (f)), links :1— 3 
and 3— 34 are formed. This typo of X 
trams when we have piles; of objects; 
for instance, in Figure A TOWER,* 
:13 and ;22 are considered as belong¬ 
ing to the name body, due to thia type 
of vertex. 

FEAK. All its regions (table IV (h)), except the one 
containing the obtuse angle, are Linked to 
«oh other fiee e 3m Figure-14 'CORN.' 

T. A search in mad# for another T to match the 
vert**- Currently analysed i two T'a match if 
they are wlknear and Taclng each other;* 1 
if there are several own, the closest ia 
chosen. For instance (Tibi* FV (d)), A and 


B are paired. An indicator 'NEXTE' ia 
plaoed in such vertices. 

(a) OclCc the NE2CTE is determined, 
EVERTICE3 eatablishe? a link be¬ 
tween :I -and 2 {Table IV(d»,and 
another between 3 and ;4. These 
hnks wiii not be produced if the re- 
euita of them is to associate the back¬ 
ground with something that is not 
the background. 

(b) The following test is done (Figure J? 
'PARALLEL*): If neither :J or 2 is 
tbr background, but both have it as 
neighbor, and in some part of the 
boundary of ;1 (end the same holds 
for ;2) with the background, the seg¬ 
ment them (M — K in Figure B 
'PARALLEL") is parallel to the cen¬ 
tral segment (0—P) of the T, then 
il and ;2 arc baked. For instance, in 
Figure 4 'STACK,' tbs analysis ap¬ 
plied to the vertex T will produce 
evidence that R3 and 14 belong to 
the same body (since EA—AA, T— 
MA Slid OA—NA fcre parallel). This 
is a ratlw global heuristic although 
only useful for bodies with parallel 
faces. Also, EVERT ICES el&wifies 
T's aceotding to the elope of their 
central Hfmata. 

A summary of the strong links put hy 
^VERTICES is found hi Table TV'GLO¬ 
BAL EVIDENCE. 1 

TJOLNT®. This; function actuates On T'a and s»* 
tabliabed global evidence as described iutxitrt (a) ofT 
(Table IV (d)). 

LGCALEVTDENCE. An arrow is a leg if one of its 
left or right vertices ia a comer (if necessary, through a 
chain of matched. Ts) which has a side parahe] to the 
cent™.] segment of the arrow. The function LEGS finds 
legs in the scene, and stores this LoJormituM Lo a list of 
'weak, 1 lirtks (see Figaro & 'LEGS'). 

GLOBAL- Strong evidence is analysed ia this part of 
the program, WbAfi thin section is entered, several links 
{strong and weak) exist among the different, regions of 
the scene. These links are shown pic-tonally inEiguic 10 
"LINK.&,' for tbe scent 'BRIDGE' (see both), AH the 
links to the background (.30) are deleted l the back¬ 
ground cannot be part of any body. 

Dc/utfmv,: a nucleus (of a body) is either a region or a 
set of nuclei which has been formed by the following 
rule. 
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TABLE IV"— 1 'GLOBAL EVIDENCE* Tbs alrcmp iinla are 
irpnKDted by dieted IIdu 




FIGURE ^'PARALLEL.' A link i*. atoblitili«J betaken -* 
*nd -2 lj«.4iAif< thr io nut to tfef UniffMl*l. but th= 

hyifgrniirnt is. k rrigklor al IhAIi of Oicm, soil LhS Bmntftft that 
W pantos !l fiern Uia hoc (ground Janet the name a true Far iij 
M puilM tu OP. tbt centnl Hitfiiant-flF Lh.t T m (martin* i» 
tbc btKkimiad). 



PIGURKT— J FfKRK. F 

Ride; ££ two nuclei art connected by two or more 
Y,n\c m theyare cnfiqeedintoalargsrnuihdW by cJjncAt*na¬ 
tion- 

(TwomudWAiaiB trttbkedif rtgwna a- and b are 
linked where & * A and b i Bj. For instance, repona -9 
and :ll are put together, because there eiiat? two Links 
among them., to form the nucleus ;8^11. Now, we see 
thatre^tt A (see Tlture 10 'LINKS') hv> two Links 
with this nucleus ll r and therefore the new nuclei 
S—11 = 4 ia formed- 

Wc let the nuclei grow and merge under the fanner 
ruk , until no new 1 nuclei. can be formed-When thliiitlH 
eftw, the scene has been partitioned into several JJ in**i- 
1 , 1 ^ 1 ” uueld; between any two of those there arc reffo or. 


^ v S 

FIGURE B—'LEGS." 

TtifiW: diffmr.-l in^« legs- 

at moat, one. link- tor example, figure 10 TJNK& will 
he transformed into-Figure 1.1 ‘NUCLEI 1 

LOCAL. If some strong link joining two “muirasl" 
nuclei ia alao peinforeed by a weak Link,, tlvesa nuclei are 
merged. 

For esample, in accne BRIDGE I' 'k 13 ™ ^ 

‘BRIDGE') the following local Luika exist {among 
others); fi to A t ilOto A, 38 to 3&, -IS to :l£- There¬ 
fore, the corresponding nyclcl are merited sjh 3 ^now 
figure NUCLEI ia transfomeid into figure 'NEW 
NUCLEI/ 

A weak link dees not- cause the regions which rt Links 
to b* merged or eftnaidfired as belonging to the same 
body uolasa there is, in add-on, one strong erHdeilW bo* 
tween such regions. LOCAL may call (■■LORA I- sgjdn, 
if necessary. 
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FIGURE |1)—'LINKS. 1 This figure Mpresenti heius 
'BRIDGE/ with tbeatrong links 'sicrArrsii its regions (hiprrjf n tei 
here by cLrehs) albs'*!!. The United Jinkr. rnp-rrew; the ssidenoe 
Ememlid hy th* v$rte* FB (see Figure L 'BRIDGE"?, The abort, 
an-Dn-a *how (Jifl Jinks pul bj.- vertex JI5; ant* l;|at * link twl-wetn 
i ■»<* PCI. bfc4ure S -free Figure $ ‘RUTDOB 1 ) i* * paw¬ 

ing t-joieiL Z*(-iAg Jinks &r* pruduocii by Uis mechanisai (te- 
accited in part <b> of T. CSee Figure S 'PARALLEL') Ctitlsd 
linki are produced by vertex GB; even if this vsr(ssr were <w- 
cluiled, and the links s*Vrt missing, there is stilt rastlgh svidsnoe 
to iurnuTv regions 1 r.5 and ;& up bdn raging to the ram# body. 

W*sk I inks an? not jtruvm. 


SINGLEBODY. A strong link joining- a nucleus and 
another nucleus composed by a single region la con¬ 
sidered enough evidence and the nuclei in question 
iTiergcd, if there is no other Link emanating from the 
single region. For instance, in Figure 13 'NEW NU¬ 
CLEI/ nucleus ; 16 is merged with nucleus :1ft—:1ft (see 
Figure i3'FINAL'). rCucleua :2B - 2ft is not joined with 
:2G—22-—23 nr with ;24—25—27— IE—21—S. Even if 
nudfliLS :2ft—2ft -were composed by a angle region, it 
atill will not be merged, since two I inks emerged from it: 
two nuclei rlisi m its possession. 

RESULTS. After (laving screened out the regions 



FIGURE LE—'Wt'CLEl.' Afler /urni-ng ill niicLoi hnviiip; twi> 
ot Irtute link* ill Kinrunun, the renrcser.ta-'.inn for Ike tr-sr,# 
'BRIDGE' I'hange* from IliBt tdiovn in figure 10 ' LINKS' to the 
une shown here. 



FIGURE 12— 1 'MW NUCLEI.' Tiur figure sTionu the Brets* 
■RPEIXiE, 1 sftjtr T.OCA1, tmss,farmi it from the re presentation 
in Figure 1L 'NUCLEI' to the one shewn here. 


that belong bo the background, the nuclei, arc printed aa 

''bodies.' 1 

Tn this process,, the Sinks which may be joining some 
of tbe nuclei are ignored: RESULTS considers the 
jinks of Figure 13 ‘FINAL 1 f for instance, As non-uaiateut. 

Summary 

SEE usch a variety of kinds? nf evidence to “link' 7 to¬ 
gether regions of a scene- The links? in SEE are supposed 
to be general enough to make SEE an ohjeet-analireis 
system. Each Imk is a piece PE ovidfituus that suggests 
that two or more regions kw from tins same object, and 
regions that got tied together hy enough evidence arc 
considered as "nuclei” of possible objects. 
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FIGURE 13 -—'FINAL . 1 SINOIJ 5 BOPY jisinta Lb- bceLy w- 
porc : 1 ft witbi Oifl Qijfl«ijt :lS— 3 fl. 


Soma examples 

Wa pxwent in thus section ac-venal acetiea with the n»- 
Bnlte s>bt&med by SEE. 

EwfiplA. 'CORN.' The program analyzes {be aw tit 
CORN (uec £gura 14 CORN 1 ) obtaining the following 
idmlideation of bodies: 

{SINGLEBODY ASSUMES (3 3 ;lj (:4j SAME 
BODY) 

(BODY 1. IS :2 :3:I :*) 

(BODY 2.1& :T A JJ} 

(BODY 3. IS :13 ill :12) 

(BODY 4, IS 45 l!& 44) Rewfa fn>r 'CORN' 
(BODY 5. IS :lld :1S 47) 

{BODY 6. IS :31:2p) 

(BODY 7. IS :B :9 :10) 

Tha region :4 got a Miig blank with 3, Mid SINGLE- 
BODY bid, to join ft with the ntielftta ;1 3 ;3. Note 
that :i and 42 did oat got joined, The pyramid at tk 
top was easily idouUGed because its psalk product* 
milky links. 

Example.'MOMO'. (S<*Flfm 15 'MOMO'). Thfeie^ 
adits are as IdUo wa: 

(LOCAL ASSUMES (47) (ft) SAME BODY) 
(LOCAL ASSUMES (ft 47} (41) SAME BODY) 
(BODY 1, IS 3 £ 4) 

(BODY 2. IS r32 33 37 :2S) 

(BODY 3, IS :2S 31) 

(BODY 4. IS il* r2t) 34 30 39) 

(BODY5.IS36 35) 

(BDDY(LIS34 3 31 A) 

(BODY 7. IS :25 33 :22) Reunite far 'MOMO 1 
(BODY 3. IS :14:13 43) 


(BODY 0. IS :10 49 41 42) 
{BODY 10. IS :18 ;9 :17) 
{BODY 11 IS :7 3) 

(BODY 12. IS 33 37 -ffl) 



FIGURE it —’CORN. 1, Bllict h bilk 1'etirorn. :* an:l ;Lli m an* 
fetavlijbed, th* htxliaa foujiii in 'hat part nf tha wane aw 
:| :2l3 14 and :I1 :l7 zlS 



FIGURE 15 —'MOMtV 
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Cv&] mtni.? ffji the aeJ c tie* far ‘MQAfQ': The central 
cubes twe easily isolated. 21 gets a link with ;5 and 
iukALlt with 24, Ed there is no need to uk LOCAL in it. 
Tlw; same is true of ;2&with :27 and 23. 

There is enough strong evidence to joint :28 and ill, 

Tluf link? for 2& 20 24 20 :19 writ $ :17 rl& arc in- 
die&ted in Figure- HJ 'MOMO-LINKS. 3 

The dolled links between regions 20 and :3d, and be¬ 
tween :90 and 09 (wefigure 15 'HOMO') ate due to the 
heuristic of parallel lames {of Figure 8 'PARALLEL'), 
These links made it possible to join the otherwise dis¬ 
connected nuclei ;29- 20- ;1Q and 24- 20. In, particular, 
if 24 or 20 did not have parallel aides, SEE would have 
failed tu merge these nuclei, and would have reported 
two bodies- The disposition of regLnna i n HOMO is such 
thftt nn link is present between :2i? and 24, since 2S 
Sfld 21 fall "exactly 11 parallel to 24 and 20. In a 1™ 
extreme scesie, HHntof these links would be present, al¬ 
lowing ooiTftcIi identification even if 3ff-:30-24-20-: 19 
were- twrt (t pram- Anyway, frEE did not make any mis* 
t*lre«- 

The triangle of links forinod by : LB and :17 nor¬ 
mally would have produced 3 separate bodies (since we 
need at least One double link to merge two regions); 
nevertheless, in this case, LOCAL makes some assump¬ 
tions and saves the situation. Again, ii :fl were not a 
jMiMl 1 clogra;n, thcre would he no weak Jinks between 2 
and :16 or between -d) and 27, and 3 bodies would have 
been reported iaatcad of ;9-:17-:ia. But we should no- 
tier that, in general, two links instead of one would be 
between ;J7 and :1&, 

Link* were established between :I2 and ;I3, without 
serious QOOSMJBUHHn, because we need two mistakes, 
two wrong strtrag links, to fool the program, But that 
could happen. 



FIGURE 1A—MOMO—LINKS', &rne Lin.Vs <xf 
Enure 15 J MOMCC 


Example. 'HOME', —(see scent- HOME). The msulta 
are in Table V' RESULTS FOR HOM E.' 


taisGLEBcmy assumes:-iho (29) same body) 
(siHGLcnonv assumes ■:3ij W) same bodyi 

(SlKGLEOODY ASSUfatH f:2S :?J :22) fiSffli SAME BODY) 

(Sisglekody assumes (aw (3i: same body> 
results 

(BODY I. IS 2 2 :10 :S1 
(BODY 2 . 13:14 :I 2 :[f : |Sj 
(BODY lISfl i-Hl 
(BODY 4. IS :J ftl 
(BODY A 13 ill 2) 

{BODY U- (3 :3& :3ft;. 

(BODY 7. (S :7 2fi :271 

(BODY BL 13 :2& 211 Results far HOME 

(BOD Y 0. [S :2ft :2S> 

(BODY lft. IS 24 23J 
(BODY II. IS 2L 22 :lEij 
(BODY 12. IS :2S 24 22 231 
{BODY IS- IS :1&> 

(BODY 14. IS lift] 

{BODY IE, IS an 
(BODY 1(5. IS :3S 37 251 

TABLE Y—Results fnr HOME 



FlfrUB-E IT—‘HOME. 1 The lesBRomal pHBItfl did CM -crKdc 
-iljffvciil tin?; SING-LILHODY was- ntcdrd tu arDd|} 24 with :3S, 
That lwdy :IJS-3B ideemied a® * 8lii*le W1B, but :l6-47 
reported u,i two. Sole ttiBt LbBW d*>W RBt evict local Link hitTraeii 
21 and. 2ft; nererihetKO, S1NQLEBOPY makes tin correct 
ttUatlflartion. (geo Table V). 


CtitKM&fto efi ft r aofedwn to HOME: There ij a certain 
deip-ee erf ambiguity In this secnc which huraftn beings 
tend tu resolve by aeeurniuK FmtwlldppiFiedflv :1G and :1T 
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arc reported as two separate bodies, instead of one, 
which is probably a more natural answer. 

In the picture from which 'HOME' was drawn, 
;19 ami rib were the two faces of a single parallelepiped 
leaning on :33-;39| SEE reports ;lfJ as one body and ;1B 
4$ another, 

Nevertheless,. SEE reports 5(3 and ;3& as forming 
pfcrt of the wune body (wltieh was. in fact the ease in tbfc 
picture in question), due to the fact that at and :l are 
background. 

EliampEe. : SPREAn.-'— (sfr Figure IS ' SPREAD T 
The result am: Ln TnU* VI 'RESULT* FOR SPREAD'. 


(LOCAL ASSUMES \ M) (4) 0 AME FM )DT| 

{LOCAL ASSUMES- i^U3/ (31 32 :M) SAME BODY 
{LOCAL A^RfMES <;1.4> (:J7j SAME BODY) 

(LOCAL ASSUMES (:5) (31) SAME BODY) 

{LOCAL ASSUMES (.3 :JI rfl) (:12 :30l SAME BODY} 
{B1N&LEHODY ASSUMES fiZm (-.ZZi BAMEBODY1 
{BIWGLEBODY .ASSUMES (4 :38l (:27) SAME BODY; 
(SIKOLtBODY ASSUMES (2S :Zfl. [ST - U2S) SAME BODY) 
RESULTS 

(BODY 1. IS 30 M :3E; 

{BODY 2. IS !2S :2Z; 

(body 3. is A-i m; 

(BODY* IS a m a?) 

(BODY IL IS 3*) 

(BODY 6, IS ilBJ :»:27 :25J Ft«uJta fai SPREAD 

(BODY 7. IS 31 at m :30> 

(BODY S, IB 49 ’A) 

(BODY*. IB :|S :3ft 3*1 $5 
(BODY ID. IB it3 :7 :IJ 
(BODY il. IS 31 3) 

(BODY JS. 10:3:33) 

(BODY 13. IB *7 :Ifl) 

(BODY U. IS AS :43 AV, 

(BODY 1ft. IB :19J 
(BODY 3fl. IS :IS 114> 

TABLE Yl—Results lor SPREAD 


Cptamerata an sht iufulHMt Co xSFKEAD: The bendy 
^2-23-24, due to Insufficiency of Links, wan split- Ln two: 
32-23 and 34. 

Since there is only one link between :0 and so. this 
body get* split into two: :G-2land ;5-20. Note that ;2I 
in nat the same face as :20, and there is where SEE gets 
confused and refuses to see evidence toward linking #1 
with ;E0. 

The tong body rS-lfl-l 1-1&-5 gC(^ properly identified. 
Eawnpk, ^ISARD/—Thifc s(se^e ie wlttlyredl with the 
following requite; 

(LOCA L ASSUMES (:11) (12) SAME BODY) 
(LOCAL ASSUMES (!£) (IS) SAME BODY) 
RESULTS 
(DODYl.lS ;I2 11) 


(BODY 2, IS :10 :15) 

(BODY 3, IS ;32 ;3l :30) 

(BODY 4. IS £ -AO *) 

(BODY o, IS :1ft :1G ;2Q) 

(BODY S. IS ;lft :l 7 Ai) results for 'HARD' 
(BODY 7. IS :5 A) 

(BODY 8. IS :1 £ 33) 

(BODY 9- IS 34 33 :22 :3 31 33 \29) 

(BODY 1- 10. IS 35 30 37) 

(BODY IRIS :7) 

(BODY 12. IS :6l 



FIGURE m— ■ SPREAD. 1 41 and 44 are Mtentifiad u i 
hii^Ik (wdy, lS*^'Ertfc*iead, gwis broken into iS-18 arid 

:1 0, riA-27-3S-2i REti eartwily idSutofod, Sd Mi m the funny 
LookinK :S9-3fl-3i-32. (S*j T>bta Yl}. 


Cdjttown(s on ike wrttifEim to HARD? :1b acid :1S hfii.'e 
to make use OF wy&k evidence to get joined end recog¬ 
nized as forming part of the same body. Nevertheless, 
tnift is n.01 necessary with :2S and :2fi, because, through 
A chain ol Ts, there is enough evideruce in 3, ;2l and ;23 
tojoinguccessfuilyslli that h>pg and twice occluded h<idy. 

There La one serioua hug in this identlhrattiun: regi-om 
:7 and :6 get identified two separate bodies, and not 
as a single one, as One would normally do. This is caused 
by the fact tlyit neither :7 nor ;C have visible ‘ueeEuF ver- 
tioem, and there are not enough parallel lines in them te 
use the heuristic of Figure 3 PARALLEL.' 
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FIGURE JB—'HARl?. r Body :2L-22-3-33-3-t-2S-2B \e rcp&rtcd 
u a sir.sli.' d'jjkM, n-jiicb n cerrecr-. Nuver!Iwl«», lifiiurs :7 end :6 
Brt repyrleU m l«cu liodi™. 

r33 was recognized as pari of 1-2, ns it should be. 
DISCUSSION 

a. 

We have desorbed a program that analyze* a three-di¬ 
mensional scene {presented in the form of a Line draw¬ 
ing) and splits it into "objects” on the basis of pure 
form. IT we consider a scene as a set of regions (sur¬ 
faces) , than EE E partitLona the set into appropriate sub- 
seta, each, subset fanning a thrOO-dimcnsional body or 
object. 

The performance of £E E shows to us that if is psssfif e 
la rtpnraU a scene inta> the objects farming tf, ws'ihoul med- 
zrejr to fciwMC fn ctetatl these objects; SEE does not need 
to know the 'definitions' or descriptions of a pyraTn id, or 
a p-e&twgonal prism, in order to isolate theseobjectH in a 
twnr wntpinln^, them, even In the ease where they are 
partially oocluded. 

The basic idea behind SEE is to make global use of in¬ 
formation collected locally at each vertex: this Informa¬ 
tion is. noisy and SEE has ways to combine many dif¬ 
ferent kiitds of unreliable evidence to make Fairly re¬ 
liable global judgments. 

Tfw cwcnllaLs are: 

(1) Representation aa vertices (with coordinates). 
Line?and regions 

{$} Types ofvertww. 

{3) Concepts of links (strong and weak), nuclei and 
rules Fpr forming them. 

The current version of SEE is restricted to scenes pre¬ 


sented in symbolic form. It does not have resources for 
dealing with curves, shadows, noise, missing lines, etc, 

it represents a “geometric theory of object identity” 
at present, 

feincc SEE requires twn strong evidenced to join two 
nuclei, it appears that its judgments will Lie In the 
' safe’ ftidt, that is, SEE will almost never join two re¬ 
gions that belong bo different bodies. From the analysis 
of Beetles shown above, its errors are almost always of 
the same type - remans that should be joirued are left 
separated- We could say that SEE beha w "conscrv- 
aHvcly," especially in the presence of ambiguities. 

Divisions of the evidence into two types, strong and 
wont, results ih a good compromise. The weak evidence 
is coneidwed to favor Linking the regions, but this evi¬ 
dence is used oniy to reinforce Evidence from more re. 
liable- clues. Indeed, the weak links that give extra 
weight to IKftHy parallel lines arc a concession to ob- 
jeel-recognition, in the sense of letting the analysis sys¬ 
tem exploit the fact that rectangular objects sue com¬ 
mon enough in the repl world to warrant special atten- 
tiort- 

Most of tbe ideas in SEE will work on curves too. 
However, we do not yet have a good idea of how sensi¬ 
tive the .system will be to "symbolic nOtae, rt Lb.j missing 
misplaced, and False region boundaries:. As indicated in 
the scenes above, this system seems to have good re¬ 
sistance to "aeoideuta" in which object® happen to 
to 4 "|Lne up" unusually well. This feature may be neces¬ 
sary iT the preprocessor that (eventually) will feed data 
SEE decides to report two nearly colinCftr lines aa one 
atone, or if it lumps several vertices into One, because 
they lie dose to each other. 

■Eufanatcitz. (None of this incorporated in the actual 
program-) More heuristics could be added to increase 
the number of links' in particular, given a good! number 
of "b&ik proposers," parameters set outside (by the 
user) would tell REE which, eel of heuristics to ueer for 
instance, if we knew that thcMChe is formed by prisma, 
we Could use the heuristic* that **k for parallel lines 
having a given configuration, we could ch*ek the length 
of certain edges, etc- 

Different kinds of links Could also be established; in 
this way . 1 contradictory" links (such as the three links of 
figure 13 'FINAL* which SEE just ignores) could be 
studied further, in order to discover ambiguities. It 
particular, a "conditional link” would be useful; regions 
:2 and ;S belong to the game body Lf region does not . 

SINGLEBODY could be unproved so (t* to analyze 
in more detail the regions that by themselves form 
bodies (that is, the bodies formed by only one region); 
in this way, wc could hope U joint regions :G and ;7 of 
hccqc 'HARD. 4 
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ADDENDUM 


The work reported here has been extended in several ways: 

handling of stereo information {two views, left and right,, of 

the sine scene), improvements to deal with noisy (imperfect) 

input, figure-hackground discrimination, optical illusions and 

some aspects of psychological perception. The interested 

reader could find these in: 

Guzman, A. Computer Recognition of Three-Dimensional 
objects in a visual scene. Ph.D. Thesis, Electrical 
Engineering Department, December I960. Avail¬ 

able also as a Project MAC Technical Report MAC-TR-59, 

AD number AD 692 200 + 

This thesis also describes an improved version Df SEEj and gives 
more exairples of scenes decomposed by SEE. 

Of course, after the bodies have been found in a scene, it 
is natural to ask what they are, to classify them: Is body 
(rd ;7 ill) a pyramid? To answer this, we introduce the concept 
of mode1 as a representation or description of classes of object 
"pyramid", "house", "triangular prism", etc. td and dt aro two 
programs that search a visual scene for instances of a given 
model {for ej^ample, find all the cubes present in such scene) , 


TD is described in: 



Guznsan, A. Scene Analysis Using the Concept cf Model* 
Report AFCRL—67Q133 * computex corpexatinn of America, 

565 Technology Square, Cambridge, MhsE, January 1967. 

AD-6 52-017. 

DT is described in; 

Gusman, A- A Primitive Recognizer of Figures in a Scene* 
Project MAC Memorandum MAC-M-342 (A.I, Vision Memo 119). 

M»I*T* January 1967* 

A general discussion [and an attempt At classification) of 
models, their properties and ways to identify objects (poly¬ 
hedral of known forms is found in* 

Guzman, A. Some Aspects of Pattern Recognition by Computer* 
M*S* Thesis, Electrical Engineering Department, M*l*T* r 
February 1967* AD-656-041, Also available as a Project 

MAC Technical Report MAC TR 17, 

DT and TD work with models and scenes made of straight lines j 

they do not handle correctly objects partially occluded, 

Bath TD and DT could search for any model (which car be 

described or constructed in certain figure description language) 

in a scene. Another approach to object recognition is to have 

specific programs to look for a specific model {or a restricted 

class of models). in this spirit, POLYBRICK is a program that 

searches for parallelepipeds in a sesne described as a line 

drawing? occluded oases are correctly handled. 

The references are: 


Guzman, A. POLY-BRICKi Adventures in the Domain of 



Parallelepipeds. Project Jv4ft,C Memorandum M?iC"fl-308 
(A.I. Memo 96}, M.I.T. May 1966. (Also found in 
MAC-TR-37,) 

Guzman, A. Object Recognition; Discovering the Parallele¬ 
pipeds in a Visual Scene. Proceedings of the Second Hawaii 
International Conference on Systems Sciences, University of 
Hawaii, Honolulu, Hawaii, January 1969. Western Periodicals 
Co. pp» 479-S2. 


NOTES: Reports with AD numbers (such as Ap-652-017) can also be 
obtained as followsx 

Government contractors Joey obtain copies of the report from 
the Defense Documentation Center, Document Service Center, 

Cameron Station, Alexandria, Va* 22314, orders will be expedited 
from DDC if placed by your librarian, or some other person author 
i-zed to request documents< 

Other U,S, citizens and organizations may obtain copies of 
this report from the clearinghouse for Federal Scientific and 
Technical Information (CFSTI), Sills Building, 5235 Port Royal Rd 
Springfield, Va. 22151, Cost: $3,00 each. 


